Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Load balancing cache in POP consumption mode #9131

Open
wants to merge 7 commits into
base: develop
Choose a base branch
from

Conversation

YoWuwuuuw
Copy link

@YoWuwuuuw YoWuwuuuw commented Jan 14, 2025

Which Issue(s) This PR Fixes

Add cache for each client request broker end rebalancing in the pop consumption mode to avoid wasting computing resources of broker end due to repeated calculation of load balancing without changes. The effect is more obvious in theory when there are a large number of queues or consumers

Fixes #9123

Brief Description

To add caching for broker-end rebalancing under the pop consumption model, use ReentrantLock to maintain consistency and efficiency. Before performing load balancing calculations, attempt to obtain cache entries when the client list and messagequeue list remain unchanged. Only if the cache is not hit will the calculation logic be executed, using minimal space to avoid wasting broker computing resources and ensuring efficient rebalancing requests

How Did You Test This Change?

Ten consumers were registered for the test consumer group, and mock 15 queues were set up to test the correctness and performance of the broker-end doLoadBalance method under the test pop consumption mode. All QueryAssignmentProcessorTest (including the original 5 methods) passed, and the performance is as follows (in the local environment)

32a8f585dfbee87e9d7e2cbdb32317a

d6e52c5a9e21da12f5038f1b1c14df1

c446c0d2f326bc8a11a36836447db2d

@3424672656
Copy link
Contributor

Hello, please fix the code running test and output the test document

@YoWuwuuuw
Copy link
Author

Hello, please fix the code running test and output the test document

okok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Load balancing cache in POP consumption mode
2 participants